Distilling Monolingual Models from Large Multilingual Transformers

Although language modeling has been trending upwards steadily, models available for low-resourced languages are limited to large multilingual such as mBERT and XLM-RoBERTa, which come with significant overheads deployment vis-à-vis their model size, inference speeds, etc. We attempt tackle this problem by proposing a novel methodology apply knowledge distillation techniques filter language-specific information from into small, fast monolingual that can often outperform the teacher model. demonstrate viability of on two downstream tasks each six languages. further dive possible modifications basic setup exploring ideas tune final vocabulary distilled models. Lastly, we perform detailed ablation study understand different components better find out what works best under-resourced languages, Swahili Slovene.